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[57] ABSTRACT 

Methods and systems are provided for generating a real time 
multimedia data stream with accurate time stamps for 
decoding. Time stamps are provided which compensate for 
a difference between a video frame rate corresponding to a 
video frame within the real time multimedia data stream and 
the oscillator clock. The video frame is then stamped with 
the time stamp which compensates for the difference 
between the theoretical presentation time stamp correspond- 
ing to the video frame and the oscillator clock. 
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METHODS AND SYSTEMS FOR ENCODING arriving video frame may then be stamped with the time 

REAL TIME MULTIMEDIA DATA which corresponds to the predicted arrival time for the that 

frame. The time stamp may be associated with the encoded 

FIELD OF THE INVENTION video information (a video presentation time stamp or 

Hie present invention relates to the field of multimedia 5 ^ and the audio information (an audio presentation 

systems. More particularly, the invention relates to the time stara P)- 

encoding of video and audio data. More data may be required to display an image than to 

generate accompanying audio as an image may have varying 

BACKGROUND OF THE INVENTION resolutions and include motion and frame rates not included 

Audio and video data has been used to supplement the 10 in the audio data * Thus » data ma y occur more fre " 
capabilities of many known systems. For example, video q uentlv Wlthia the MPEG data stream - ^ infrequent inter- 
data has been used to supplement the capability of the spersion of audio data between video data may cause an 
telephone to produce a multimedia system which may be ima S e frame to be displayed before or after the audio has 
referred to as a teleconferencing system. The teleconferenc- been reproduced. The time stamps may be used to synchro- 
ing system provides video data to supplement the audio data. 15 nize the Presentation of audio and video at the decoder 150. 
The inclusion of video data in a multimedia system, In general, the audio and video data may be synchronized 
however, may increase the likelihood of data bandwidth by the decoder 150 by comparing the audio and video 
problems. Specifically, the size of the audio and video data presentation time stamps with the system clock reference 
may make it difficult for the system to transmit all of the 2Q (SCR) 125. The SCR 125 may indicate how much time has 
required data in the necessary time. Consequently, some elapsed since the start of the multimedia data stream. Ideally, 
multimedia systems utilize a form of encoding and decoding the time stamps within the multimedia data stream corre- 
of the audio and video data to reduce the required band- spond to the SCR 125. By comparing the SCR 125 with the 
width. Several multimedia specification committees have presentation time stamps associated with audio and video 
established and proposed standards for encoding and decod- data, the decoder 150 may determine whether the video and 
ing audio and video information. MPEG1 and MPEG2 (the 25 audio data are leading or lagging SCR 125. The decoder 150 
MPEG standard), established by the Motion Picture Experts may then compensate for the lead or lag. For example, if the 
Group, are examples of such standards. decoder 150 determines that the presentation time stamps 

A system using the MPEG standard may compress real are lagging the SCR 125, the decoder 150 may discard the 

time audio and video data for transmission over a network 30 data associated with the lagging presentation time stamps, 

where the audio and video may be decompressed and FIG. IB illustrates an exemplary arrangement of video 

reproduced. The system may compress each video frame to frames, corresponding SCR times and the presentation time 

reduce the amount of data required to reproduce the equiva- stamps associated with the video frames. SCR times 33 ms 

lent frame for display. Video frames may be compressed in through 166 ms correspond to the arrival times for video 

three ways according to the MPEG standards. An intra or 35 frames provided at a particular theoretical frame rate. Frame 

I-type frame may comprise a frame of video data coded numbers #1 through #5 correspond to the encoding time of 

using information about itself. One given non-compressed video frames provided by the real time video source. PTS 

video frame may be encoded into one I-type frame of times 33 ms through 166 ms indicate the time stamp on the 

encoded video data. A predictive or P-type frame may corresponding video frame. For example, video frame #2 is 

comprise a frame of video data encoded using motion 40 encoded at SCR time 66 ms and is stamped with a PTS of 

compensated prediction from a past frame. A previously 66 ms. 

encoded frame such as I-type or P-type may be used to As multimedia systems proliferate consumer electronic 

encode a current non-compressed frame of video data into a devices may be used to supply audio and video input. For 

P-type frame of encoded video data. A bi-directional or example, a multimedia system may include a video cassette 

B-type frame may comprise a frame of video data encoded 45 recorder (VCR) to supply realtime video input. The encoder 

using a motion compensated prediction from a past and 100 could then use the frame rate specified in the MPEG 

future reference frame, or a past, or a future reference frame header to time stamp the associated video and audio data, 

of video data. A reference frame may be either an I-type There may, however, be problems associated with the use of 

frame or a P-type frame. If a reference frame were to be lost various consumer electronic devices. For example, some 

or discarded, the decoder may slow down when decom- 50 consumer electronic devices may produce video frames at a 

pressing subsequent video frames which were compressed rate which is not in accord with the specified rate. Similarly, 

with reference to the discarded frame. the video output provided by the consumer electronic device 

FIG. 1A is a block diagram of a realtime multimedia may not be in accord with known standards associated with 

system using encoding and decoding. The encoder 100 may video signals. In other words, the specified video frame rate 

accept analog audio and video information to produce a 55 may be inaccurate, causing video frames to be stamped 

multimedia data stream which may be transmitted across a inaccurately. Moreover, many consumer electronic devices 

connection 120. The decoder 150 may then decode the may produce a comparatively low quality video output. For 

multimedia data stream for presentation to the end user. example, some spurious signals may occur in the video 

The analog audio and video information may be pro- supplied by the consumer electronic device which may lead 

cessed by encoder 100 using video encoder 105 and audio 60 10 an inaccurate frame rate. 

encoder U0. The video encoder 105 and the audio encoder FIGS. 2 and 3 illustrate how video frames may be 

110 compress the video and audio information according to stamped with inaccurate time stamps when the specified 

the MPEG 1 or MPEG2 standard. A specified frame rate may frame rate is incorrect. FIG. 2 illustrates the time stamps 

be used to predict the arrival time of each video frame from associated with encoded video frames when a real time 

the real time video source. For example, if the specified 65 video source provides video frames faster than the specified 

frame were 30 frames per second, a prediction may be made rate. A faster than specified frame rate may cause video 

that a video frame will arrive approximately every 33 ms. An frames to be time stamped with times which do not corre- 
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spond to SCR 125. For example, frame #1 is encoded and It is another object of the present invention to reduce the 

stamped with a PIS of 33 ms. Frame #2, which according cost of a system for encoding and decoding, audio and video, 
to the specified video frame rate should be encoded at It is yet another object of the present invention to produce 

approximately 66 ms, is encoded some time earlier. Despite an om t which ^ decoded b a standard decoder> 
the actual encoding time, frame #2 is stamped with a PTS of 5 ™. , . ,. t e t . 

66 ms. Frame #3, which according to the specified video ^ and other objects of the present invention are 
frame rate should be encoded at approximately 100 ms, is P rovided b ? determining a time stamp which compensates 
encoded some time earlier and time stamped with a PTS of for a difference between a video frame rate corresponding to 
100 ms. Frame #4, which according to the specified video a Vlde0 frame ^thin the real time multimedia data stream 
frame rate should be encoded at approximately 133 ms, is and the oscillator clock. The video frame is stamped with a 
stamped with a PTS of 133 ms. Frame #5, which according 10 compensating time stamp which compensates for the differ- 
to the specified video frame rate should be encoded at ence between the theoretical presentation time stamp corre- 
approximately 166 ms, is encoded at an SCR time of 133 ms spending to the video frame and the oscillator clock, 
and stamped with a PTS of 166 ms. In other words, frame The compensating time stamp may be generated by 
#5 should have been encoded at approximately an SCR time comparing the oscillator clock to the theoretical PTS. If the 
of 166 ms because the specified frame rate indicates that the 15 difference exceeds a first predetermined threshold value the 
fifth video frame should be encoded at that time. Frame 45, compensation may take place in a coarse adjustment mode, 
however, is actually encoded at an SCR time of 133 ms (i e If the difference ^ less than or e al t0 a ^nd predeter- 
a full frame time earlier than it should have been encoded) mined thfeshold the compensation may ^ made J a fine 

K J! ^ n ^S^Jt^T^ " ,n «8^ent mode. Once the compensation is determined, the 

loo ms (i.e., an inaccurate time stamp). 20 f a. u *■ * j j- . j 

rr. \ A . , ... j ,1 • video frame may then be time stamped with the adjusted 

The situation described above may produce problems in pjg J 

decoding the actual video frames. When the video frames . . ,. 

are produced at a rate which is faster than that specified, the* u Fine adjustment may be accomplished by multiplying the 

decoder's 150 input buffer may be completely filled. The theoretica presentation time stamp by a first compensation 

decoder 150 may then discard a number of frames to catch 25 factor multiplying the oscillator clock by a second 

up which may cause the display to appear discontinuous. compensation factor The two products may then be added to 

Moreover, if any of the discarded frames are reference P r0Vlde the adjusted FrS. 

frames, decoding subsequent frames may be delayed which Coarse adjustment may be accomplished by incrementing 
may cause additional display artifacts the theoretical PTS and comparing the incremented value 
FIG. 3 shows the time stamps associates with video 30 with the present value of the oscillator clock. If the corn- 
frames when the video frames are provided at a frame rate P^ 1500 indicates that the difference exceeds a third prede- 
which is slower than the specified rate. SCR times 33 ms termined threshold value, time is added or subtracted 
through 200 ms represent the time stamps normally associ- to/from the theoret ical PTS. Alternatively, a coarse mode 
ated with video frames provided at the specified rate. PTS ad J^tment may be made by adjusting the oscillator clock so 
times 33 ms through 166 ms represent the time stamps 35 as to be equal to value of the closest in time increment 
associated with encoded video frames. Frame #1 though ot the theoretlcal VYS - 

frame #5 represent the actual encoding time for the corre- Consumer electronic devices may provide video and 

sponding frame. If the video frames are provided less audl ° data at a rate which does DOt correspond to the rate 

frequently than the specified rate, the decoder 150 may specified in a header associated with the multimedia data 

experience difficulty in decoding the actual video frames. ^ stream. The decoder may expect video frames to be deliv- 

For example, frame #1 is encoded at approximately SCR ered at the sP ecified rate > yet the actual video rate may vary, 

time 33 ms and stamped with a PTS of 33 ms. Frame #2 is Thi& mav result in -unpleasant visual artifacts in the decoded 

encoded some time after SCR time 66 ms yet stamped with videa Vlsual art ikcts may be lessened by reducing the 

a PTS of 66 ms. Because of the difference between the actual number of encoded video frames that may be discarded by 

frame rate and the specified frame rate, as time passes the « the decoder. The MPEG standard allows for certain frames 

actual frames arriving increasingly later than the theoretical t0 1x5 encoded with reference to other encoded frames. This 

time. For example, frame #5 is encoded at approximately means thal frames may be decoded by using previ- 

SCR time 200 ms, the frame is stamped with a PTS of 166 ouslv dec o d ed frames. If, however, the previous video 

ms. The PTS is for frame #5 is therefore inaccurate. frames are discarded by the decoder the video frames may 

The inaccurate PTS, may cause difficulty for the decoder 50 appear distorted or may be skipped. The present invention 

150. For example, the decoder 150 may discard frame #4 ma y reduce me number of frames that mav be d * ca "te d by 

because the PTS of 133 ms indicates that the time for ad J ust ing the PTS so that fewer video frames are time 

displaying frame #4 has passed in relation to SCR time. In Ramped such that the video frames are discarded by the 

other words, frame #4 is stamped with a PTS which indi- decoder - 

cates that the time for displaying frame #4 has passed. 55 System cost may be reduced by reducing the need for 
Consequently, the decoder 150 may discard frame #4 and special interface hardware to compensate for the inaccurate 
resort to displaying frame #3. When frame #5 is decoded, the frame rates P rodu ced by some consumer electronic devices. 
PTS of 166 ms indicates that frame #5 is too late as well and ^ present invention may address the inaccuracy by pro- 
may be discarded. Again the decoder 150 may resort to v i d ing an adjusted PTS which is less dependent upon the 
displaying frame #3. This type of behavior may result in 60 actuaI frame rate of tne video from the consumer electronic 
detectable visual artifacts such as repeating and skipping device. 

subsequent video frames. A standard encoded output may be provided by comply- 

cn w MADV ~ v TOC .xrwnKrrw,™ m S ^th the required encoding (i.e., MPEG 1 or MPEG 2). 

SUMMARY OF THE INVENTION m preseQt mvention may prov > de fl muItimedia data stream 

In .view of the above discussion, it is an object of the 65 which complies with the encoding standard and, therefore, 

present invention to reduce visual artifacts in the decoded may allow the system to utilize a decoder which complies 

video when video and audio data are supplied in real time. with the encoding standard. For example, the system may 
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employ an encoder which utilizes the present invention and specified in the flowchart block or blocks. The computer 

a decoder which complies with the MPEG1 or MPEG2 program instructions may be executed by a processor to 

encoding standard. Consequently, the decoder may not be cause a series of operational steps to be performed by the 

required to understand how the encoder provides the mul- processor to produce a computer implemented process such 

timedia data stream. 5 mat mc instructions which execute on the processor provide 

As will further be appreciated by those of skill in the art, s ' e P? f° r ™P^enting the functions specified in the flow- 

. . i i_ j ■ j , i chart block or blocks, 

the present invention may be embodied as a methods, . e . n 

on« a „t„c/c„c. m c ^ ^'.^ _ , Accordingly, blocks of the flowchart illustrations support 

apparatus/systems or computer program products. . - - r . . V K , 

r r r o r combinations of means for performing the specified 

BRIEF DESCRIPTION OF THE DRAWINGS io funclions > combinations of steps for performing the speci- 
fied functions and program instruction means for performing 

FIG. 1A is a block diagram of a real time multimedia the specified functions. It will also be understood that each 

system using encoding and decoding; block of the flowchart illustrations, and combinations of 

FIG. IB illustrates the timing of an exemplary encoding blocks 1 in the flowchart illustrations, can be implemented by 

and decoding process- 15 s P ecial Purpose hardware-based systems which perform the 

™^ ^ m. . ' . • , , , - « specified functions or steps, or combinations of special 

FIG. 2 illustrates the timing of decoding of encoded video purpose hardware and computer instructions, 

frames presented at a rate which is faster than the rate FIG. 4 is a block diagram of a realtime multimedia system 

specified by a header; utilizing the present invention. Realtime analog video data is 

FIG. 3 illustrates the timing of decoding of encoded video presented to the video analog-to-digital converter 405. The 

frames presented at a rate which is slower than the rate 20 video analog-to-digital converter 405 converts the realtime 

specified by a header; analog video data to a digitized bit stream. The video 

FIG. 4 is a block diagram of a real time multimedia encoder 415 converts the digitized video bit stream to an 

system using encoding and decoding utilizing the present cnco ^ video data stream in compliance with the MPEG 

invention* standard. The theoretical frame counter 425 uses the speci- 

, 25 fied frame rate to determine the theoretical presentation time 

FIG. 5 illustrates the decoding of encoded video frames stamp ( PTS ) 427 . For example, a specified frame rate of 30 

utilizing the present invention which are presented at a rate framcs per (fp S ) correspond to intervals of 33 ms 

which is faster than the specified rate; between each frame. The theoretical PTS 427 indicates the 

FIG. 6 illustrates the decoding of encoded video frames theoretical times at which particular video frames supplied 

utilizing the present invention which are presented at a rate 30 at the specified frame rate should be encoded. For example, 

which is slower than the specified rate; at 30 fps, frame #1 would theoretically be encoded at 

FIG, 7 is a flow chart of the operation of one embodiment approximately 33 ms, frame #2 would be encoded at 

of the present invention; approximately 66 ms, frame #3 would be encoded at 

FIG. 8 is a flow chart of the operation of an alternative approximately 100 ms, and frame #4 would be encoded at 

embodiment of the present invention; 35 approximately 133 ms. 

ttt^ n • n u * p.u *■ p it, . ,• Video encoder 415 provides an end-of-field (EOF) inter- 

HG. 9 is a flow chart ot the operation of another embodi- 4 r ,. . ,- A ™ r,^- .\ \ . + , 

. c ,u . ■ 1 nipt 416 to the PTS adjuster 430. The EOF interrupt 416 

ment of the present invention; and . . . « , 7 F 7 

indicates that the video encoder 415 has detected a complete 

FIG. 10 is a flow chart of the operation of another framc of video from lhe ^ Q analog-to digital converter 

alternative embodiment of the present invention. 4Q 405 . The oscillator 445 generates an oscillator clock 440. 

DETAII ED DESCRIPTION OF THE ^ e osc ^ ator dock 440 indicates the elapsed time since the 

INVENTION start of lne mummedia data stream. The PTS adjuster 430 

examines the difference between the theoretical PTS 427 

The present invention now will be described more fully and the oscillator clock 440. The difference indicates 

hereinafter with reference to the accompanying drawings, in 45 whether the video frames are being provided faster or slower 

which preferred embodiments of the invention are shown. than the specified rale. The PTS adjuster 430 uses these two 

This invention may, however, be embodied in many different indications to generate an adjusted PTS 432 which may 

forms and should not be construed as limited to the embodi- compensate for the difference. For example, if the specified 

ments set forth herein; rather, these embodiments are pro- frame rate is such that a video frame should be produced 

vided so that this disclosure will be thorough and complete, 50 approximately every 33 ms and the first theoretical PTS 

and will fully convey the scope of the invention to those occurs at approximately 100 ms, the second theoretical PTS 

skilled in the art. Like numbers refer to like elements will occur at approximately 133 ms. Similarly, the third 

throughout. As will be appreciated by one of skill in the art, theoretical PTS will occur at approximately 166 ms. 

the present invention may be embodied as methods or Consequently, if the oscillator clock is equal to 140 ms 

devices. Accordingly, the present invention may take the 5S during the second video frame, the adjusted PTS will be 

form of an entirely hardware embodiment, an entirely soft- equal to 133 ms. 

ware embodiment or an embodiment combining software The TS Mux 435 multiplexes encoded video data from the 

and hardware aspects. video eoco der 415 and encoded audio data from the audio 

The present invention is also described with respect to encoder 420 into a realtime multimedia data stream. The 

FIGS. 7 through 10 which are flowchart illustrations of 60 realtime multimedia data stream also contains presentation 

embodiments of the present invention. It will be understood time stamps which indicate the PTS for each corresponding 

that each block of the flowchart illustrations, and coinbina- frame of encoded video data and audio data. Tlie PTS 

tions of blocks in the flowchart illustrations, can be imple- corresponding to encoded video data is generated from the 

mented by computer program instructions. These program adjusted PTS 432. The encoded audio data is time stamped 

instructions may be provided to a processor to produce a 65 according to the oscillator clock 440. The lime stamping 

machine, such that the instructions which execute on the may enable the decoder to synchronize the presentation of 

processor create means for implementing the functions audio and video data to the user. 
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The PTS adjuster 430 will now be described in greater 
detail. The PTS adjuster 430 creates an adjusted PTS 432 by 
determining the difference between the theoretical PTS 427 
and the oscillator clock 440. The adjusted PTS 432 is then 
used to time stamp the corresponding video data within the 
encoded video data stream. The PTS adjuster 430 first 
determines whether the difference between the theoretical 
presentation time stamp 427 and the oscillator clock 440 
requires a coarse adjustment or a fine adjustment. This may 
be done by comparing the difference to a first predetermined 
threshold value. If the difference exceeds the predetermined 
threshold, a coarse adjustment mode is selected. In one 
embodiment, the predetermined threshold may be about one 
frame time at the specified rate. For example, at 30 fps, one 
frame time would be approximately 33 ms. A coarse adjust- 
ment may be required in such a case because a difference of 
more than one frame time may produce a visible artifact 
more quickly than a difference of less than one frame time. 
Consequently, a coarse mode is used to compensate more 
quickly. If the required adjustment is less than or equal to the 
predetermined threshold value a fine adjustment mode is 
selected. 

A fine mode adjustment may be accomplished by first 
adjusting the theoretical presentation time 427, adjusting the 
oscillator clock 440, and then adding the adjusted theoretical 
presentation time and the adjusted oscillator clock time to 
one another. The adjustment to the theoretical presentation 
time 427 may be accomplished by multiplying the theoreti- 
cal presentation time 427 by a first compensation factor. In 
the preferred embodiment the first compensation factor may 
be equal to 0.9. However, a first compensation factor of from 
about 0.5 to about 1.0 may be used. The adjustment to the 
oscillator clock may be accomplished by multiplying the 
oscillator time by a second compensation factor. In the 
preferred embodiment the second compensation factor may 
be equal to 0.1. However, a second compensation factor of 
from about 0.0 to about 0.5 may be used. Then, as described 
above, the two products are added together to generate the 
fine adjustment to the adjusted PTS 432. 

A coarse adjustment mode may be accomplished by 
incrementing the theoretical PTS 427 and subtracting the 
incremented theoretical PTS 427 from the oscillator clock 
440. The difference between the incremented theoretical 
PTS 427 and the oscillator clock 440 is truncated so as to 
preserve any integer component. For example, if the differ- 
ence between the incremented theoretical PTS 427 and the 
oscillator clock 440 equals 1.5 frame times, the integer 
component (i.e., 1) is added to the Theoretical PTS 427. 
Similarly, if the difference between the incremented theo- 
retical PTS 427 and the oscillator clock 440 equals -1.5 
frame times, the integer component (i.e., 1) is subtracted 
from the Theoretical PTS 427. 

In another embodiment, a coarse adjustment may also be 
provided by rounding the oscillator clock to the nearest 
theoretical PTS to produce the adjusted PTS 432. The TS 
Mux 435 then time stamps the encoded video data with the 
adjusted PTS 432 and time stamps the encoded audio data 
with the oscillator clock 440. 

These methods may occasionally produce two frames 
with the same PTS. This may compensate for differences 
between the video frame rate and the oscillator and may be 
more acceptable than discarding the second frame. The 
second frame may maintain buffer fullness which may 
improve system performance by minimizing the disruption 
of the decoder's predictive decompression. 

FIG. 5 illustrates the decoding of encoded video frames 
utilizing the present invention which are presented faster 
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than the specified rate. Oscillator clock times 33 ms through 
133 ms represent times at which time stamps are inserted 
into the realtime multimedia data stream. Video frames #1 
through #5 represent encoded video frames provided faster 

5 than the specified frame rate. PTS times 33 ms to 133 ms 
represent the Adjusted PTS 432 stamped on the correspond- 
ing video frame. A system utilizing the present invention 
may avoid the problems associated with existing systems by 
adjusting the time indicated by the oscillator clock so as to 
0 be equal to the value of the closest in time increment of the 
theoretical PTS 427. For example, frame #3 is encoded 
sometime after oscillator clock time 6 ms. According to the 
present invention, frame #3 is assigned a PTS of 100 ms. 
Frame #4 is encoded sometime after oscillator clock time 
5 100 ms and is assigned a PTS of 100 ms. In other words, 
frames #3 and #4 are assigned the same PTS. Consequently, 
frame #5 may be assigned a PTS of 133 ms which synchro- 
nizes frame #5 to the theoretical PTS for frame #5. 

The present invention, therefore, may avoid the problem 

20 found in existing systems shown in FIG. 2. Rather than 
allowing the decoder's 150 input buffer to fill, the decoder 
150 may discard some frames which are stamped with 
redundant PTS times on arrival. This way, the present 
invention reduces the likelihood that the decoder's 150 input 

25 will fill up. 

FIG. 6 illustrates the decoding of encoded video frames 
utilizing the present invention which are presented slower 
than the specified rate. Oscillator clock times 33 ms through 
200 ms represent times at which time stamps are inserted 

30 into the realtime multimedia data stream. Video frames #1 
through #5 represent encoded video frames provided slower 
than the specified frame rate. PTS times 33 ms to 133 ms 
represent the Adjusted PTS 432 stamped on the correspond- 
ing video frame. A system utilizing the present invention 

35 may avoid the problems associated with existing systems by 
adjusting the time indicated by the oscillator clock so as to 
be equal to the value of the closest in time increment of the 
theoretical PTS 427. The present invention therefore reduces 
the problem described with reference to FIG. 3. For 

40 example, in FIG. 3, the decoder may exhibit a skip and 
repeat behavior because the actual video frames would 
arrive with time stamps which are outdated. FIG. 6, 
however, illustrates that the slow frames will be time 
stamped according to the nearest oscillator clock time. For 

45 example, frame #3 may be stamped with a PTS of 100 ms 
while frame #4 may be stamped with a PTS of 166 ms (i.e., 
frames #3 and #4 are stamped with PTS values more than 
one frame time apart). Consequently, frame #5 may be 
stamped with a PTS of 200 ms which is synchronized to the 

50 theoretical PTS for frame #5. 

FIG. 7 is a flowchart of the operation of one embodiment 
of the present invention. Processing begins in step 705. The 
difference between the theoretical presentation time stamp 
and the oscillator clock is examined (block 710). If the 

55 difference between the theoretical presentation time stamp 
and the oscillator clock exceeds a first predetermined thresh- 
old value, then a coarse adjustment mode is selected (block 
725). In the preferred embodiment, the first predetermined 
threshold may be about 1 frame time. If, however, the 

60 difference between the theoretical presentation time stamp 
and the oscillator clock is less than the first predetermined 
threshold, a fine adjustment mode is selected (block 720). 
The selected adjustment mode is then executed (block 727) 
and processing ends for this frame. 

65 FIG. 8 is a flowchart of an embodiment of the present 
invention according to the coarse adjustment mode. Coarse 
mode adjustment mode starts in block 805. A digitized video 
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frame is received (block 810). The oscillator clock is then stamping the video frame within the real time multi- 
read (block 815) and an adjusted PTS is created for the media data stream with the adjusted presentation 
current time stamp by rounding the oscillator clock to the time stamp. 

nearest theoretical PTS (block 820). The TS Mux then 2. The method of claim 1, wherein said step of selectively 

stamps the current video frame with the adjusted PTS (block 5 adjusting the oscillator clock so as to provide an adjusted 

825). presentation time stamp comprises the steps of: 

FIG. 9 illustrates an embodiment utilizing the present determimng the difference 5etween the theoretical pre- 

invention according to an alternative coarse adjustment . . * ■ ... . , . r 

. r* a- * * i ■ • * ui i nnc r- * sentation tune stamp and the oscillator clock; 

mode. Coarse adjustment mode begins at block 905. First, v ' 

the theoretical FFS is incremented by the period correspond- Q selecting a coarse adjustment mode if the difference 

ing to the theoretical frame rate or frame time (block 910). between the theoretical presentation time stamp and the 

The incremented theoretical PTS is subtracted from the oscillator clock is more than a first predetermined 

oscillator clock (block 915) providing a difference com- threshold value; 

prised of an integer and a fraction. If the difference is selecting a fine adjustment mode if the difference between 

negative, the incremented theoretical PTS leads the oscilla- tne theoretical presentation time stamp and the oscil- 

tor clock (block 920) and the integer portion of the differ- lator clock ^ less tnan or equal the first predetermined 

ence is subtracted from the incremented theoretical PTS to threshold value- and 

provide the adjusted PTS (block 930). If the difference is ' 

positive, the incremented theoretical PTS lags the oscillator adjusting the presentation time stamp in accordance with 

clock (block 920) and the integer portion of the difference is lhe selected mode - 

added to incremented theoretical PTS to provide the 20 3. The method of claim 2, wherein said step of selecting 

adjusted PTS (block 935). A determination is then made as a adjustment mode comprises the step of adjusting 

to whether more frames require processing (block 940). If the time indicated by the oscillator clock so as to be equal 

more frames do require processing, processing returns to to the value of the closest in time increment of the theoretical 

block 910. If no frames require processing, processing ends. presentation time stamp. 

FIG. 10 is a flowchart of the operation of another embodi- 25 4 - Tne method of claim 2, wherein said step of selecting 

ment of the present invention. Specifically, FIG. 10 is an a coarse adjustment mode comprises the steps of: 

embodiment of a fine mode adjustment utilizing the present incrementing the theoretical presentation time stamp; 

invention. The fine mode adjustment starts at block 1005. comparing the incremented theoretical presentation time 

The theoretical presentation time is adjusted by multiplying st witfa the oscillator clock and 

the theoretical presentation time by a first predetermined ™ . . . , , 

compensation factor (block 1010). The oscillator clock is adjusting the incremented theoretical presentation time 

then adjusted by multiplying the oscillator clock by a second stamp. 

predetermined compensation factor.l (block 1015). The 5. The method of claim 4, wherein said step of comparing 

adjusted theoretical presentation time and the adjusted oscil- comprises subtracting the incremented theoretical presenta- 

lator clock are then added (block 1020) to provide an 35 Uon time stam P from the orator clock to provide an 

adjusted theoretical presentation time for use as the time integer value. 

stamp for the current video frame. 6 * llie method of claim 5 » wherein said step of adjusting 

Although the present invention is described with refer- the incremented theoretical presentation time stamp com- 

ence to the MPEG1 and MPEG2 standards, it should be pnses the steps ot: 

understood that the present invention may also be practiced 40 adding the integer value to the incremented theoretical 

in systems which implement other techniques for encoding presentation time stamp if the subtraction results in a 

and decoding video and audio data. positive integer; and 

In the drawings and specification, there have been dis- subtracting the integer value from the incremented theo- 

closed typical preferred embodiments of the invention and, retical presentation time stamp if the subtraction results 

although specific terms are employed, they are used in a 45 10 a negative integer. 

generic and descriptive sense only and not for purposes of 7 * Th e method of claim 2, wherein said step of selecting 

limitation, the scope of the invention being set forth in the a fine adjustment mode comprises the steps of: 

following claims. adjusting the theoretical presentation time stamp; 

That which is claimed is: adjusting the oscillator clock time; and 

1. A method of generating an encoded real time multi- 50 ad ding the adjusted theoretical presentation time stamp 

media data stream comprising the steps of: and the ad j us ted oscillator clock time, 

determining a time stamp which compensates for a dif- 8. The method of claim 7, wherein said step of adjusting 

ference between a video frame rate corresponding to a the theoretical stamp comprises the step of multiplying the 

video frame within the real time multimedia data theoretical presentation time stamp by a first compensation 

stream and an oscillator clock; and 55 factor, 

stamping the video frame with the time stamp which 9. The method of claim 8, wherein the first compensation 

compensates for the difference between the theoretical factor equals about 0.9. 

presentation time stamp corresponding to the video 10. The method of claim 7, wherein said step of adjusting 
frame and the oscillator clock, wherein said steps of the oscillator clock time comprises the step of multiplying 
determining a time stamp and stamping the video frame 60 the oscillator clock time by a second compensation factor, 
comprises the steps of: 11. The method of claim 7, wherein the second com pen- 
comparing the oscillator clock to a theoretical presen- sation factor is about 0.1. 

tation lime stamp; 12. The method of claim 2, wherein the first predeter- 

selectively adjusting the oscillator clock so as to pro- mined threshold value is about equal to one frame time of 

vide an adjusted presentation time stamp based upon 65 the specified frame rate. 

the theoretical lime stamp and the oscillator clock; 13. A system for generating an encoded real lime multi- 

and media data stream comprising: 
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means for determining a time stamp which compensates 
for a difference between a video frame rate correspond- 
ing to a video frame within the real time multimedia 
data stream and an oscillator clock; and 

means for stamping the video frame with the time stamp 
which compensates for the difference between the 
video frame rate corresponding to the video frame and 
the oscillator clock, wherein said means for determin- 
ing a time stamp and stamping the video frame com- 
prises: 

means for comparing the oscillator clock to a theoreti- 
cal presentation time stamp; 

means for selectively adjusting the oscillator clock so 
as to provide an adjusted presentation time stamp 
based upon the theoretical time stamp and the oscil- 
lator clock; and 

means for stamping the video frame within the real 
time multimedia data stream with the adjusted pre- 
sentation time stamp. 

14. The system according to claim 13, wherein said means 
for selectively adjusting the oscillator clock so as to provide 
an adjusted presentation time stamp comprises: 

means for determining the difference between the theo- 
retical presentation time stamp and the oscillator clock; 

means for selecting a coarse adjustment mode if the 
difference between the theoretical presentation time 
stamp and the oscillator clock is more than a first 
predetermined threshold value; and 

means for selecting a fine adjustment mode if the differ- 
ence between the theoretical presentation time stamp 
and the oscillator clock is less than or equal to the first 
predetermined threshold value. 

15. The sys)tem of claim 14, wherein the first predeter- 
mined threshold value is about equal to one frame time of 
the specified frame rate. 

16. The system according to claim 14, wherein said means 
for selecting a coarse adjustment mode comprises means for 
adjusting the time indicated by the oscillator clock so as to 
be equal to the value of the closest in time increment of the 
theoretical presentation time stamp. 

17. The system according to claim 14, wherein said means 
for selecting a coarse adjustment mode comprises: 

means for incrementing the theoretical presentation time 
stamp; 

means for comparing the incremented theoretical presen- 
tation time stamp with the oscillator clock; and 

means for adjusting the incremented theoretical presen- 
tation time stamp. 

18. The system of claim 17, wherein said means for 
comparing comprises means for subtracting the incremented 
theoretical presentation time stamp from the oscillator clock 
to provide an integer value. 

19. The system according to claim 17, wherein said means 
for adjusting the incremented theoretical presentation time 
stamp comprises: 

means for adding the integer value to the incremented 
theoretical presentation time stamp if the subtraction 
results in a positive integer; and 

means for subtracting the integer value from the incre- 
mented theoretical presentation time stamp if the sub- 
traction results in a negative integer; increasing the 
value of the incremented theoretical presentation time 
stamp. 

20. The system according to claim 14, wherein said means 
for selecting a fine adjustment mode comprises means for: 

means for adjusting the theoretical presentation time 
stamp; 
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means for adjusting the oscillator clock time; and 
means for adding the adjusted theoretical presentation 
time stamp and the adjusted oscillator clock time. 

21. The system according to claim 20, wherein said means 
for adjusting the theoretical presentation time stamp com- 
prises means for multiplying the theoretical presentation 
time stamp by a first compensation factor. 

22. The system according to claim 21, wherein the first 
compensation factor equals about 0.9. 

23. The system according to claim 20, wherein said means 
for adjusting the oscillator clock time comprises means for 
multiplying the oscillator clock time by a second compen- 
sation factor. 

24. The system according to claim 23, wherein the second 
compensation factor is about 0.1. 

25. A computer program product for generating an 
encoded real time multimedia data stream comprising: 

a computer-readable storage medium having computer- 
readable program code means embodied in said 
medium, said computer-readable program code means 
comprising: 

computer readable program code means for determining a 
time stamp which compensates for a difference 
between a video frame rate corresponding to a video 
frame within the real time multimedia data stream and 
an oscillator clock; and 

computer readable program code means for stamping the 
video frame with the time stamp which compensates 
for the difference between the video frame rate corre- 
sponding to the video frame and the oscillator clock, 
wherein computer readable program code means for 
determining a time stamp and stamping the video frame 
comprises: 

computer readable program code means for comparing 
the oscillator clock to a theoretical presentation time 
stamp; 

. computer readable program code means for selectively 
adjusting the oscillator clock so as to provide an 
adjusted presentation time stamp based upon the 
theoretical time stamp and the oscillator clock; and 
computer readable program code means for stamping 
the video frame within the real time multimedia data 
stream with the adjusted presentation time stamp. 

26. The method of claim 25, wherein said computer 
readable program code means for adjusting the oscillator 
clock so as to provide an adjusted presentation time stamp 
comprises: 

computer readable program code means for determining 
the difference between the theoretical presentation time 
stamp and the oscillator clock; 

computer readable program code means for selecting a 
coarse adjustment mode if the difference between the 
theoretical presentation time stamp and the oscillator 
clock is more than a first predetermined threshold 
value; 

computer readable program code means for selecting a 
fine adjustment mode if the difference between the 
theoretical presentation time stamp and the oscillator 
clock is less than or equal to the first predetermined 
threshold value; and 

computer readable program code means for adjusting the 
presentation time stamp in accordance with the selected 
mode. 

27. The computer program product according to claim 26, 
wherein the first predetermined threshold value is about 
equal to one frame time of the specified frame rate. 
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28. The computer program product according to claim 26, 
wherein said computer readable program code means for 
selecting a coarse adjustment mode comprises the step of 
adjusting the time indicated by the oscillator clock so as to 
be equal to the value of the closest in time increment of the 5 
theoretical presentation time stamp. 

29. The computer program product according to claim 26, 
wherein said computer readable program code means for 
selecting a coarse adjustment mode comprises: 

computer readable program code means for incrementing 30 

the theoretical presentation time stamp; 
computer readable program code means for comparing 

the incremented theoretical presentation time stamp 

with the oscillator clock; and 
computer readable program code means for adjusting the 

incremented theoretical presentation time stamp. 

30. The computer program product according to claim 29, 
wherein said computer readable program code means for 
comparing comprises means for subtracting the incremented 2Q 
theoretical presentation time stamp from the oscillator clock 

to provide an integer value. 

31. The method of claim 30, wherein said computer 
readable program code means for selecting a fine adjustment 
mode comprises: 25 

computer readable program code means for adjusting the 

theoretical presentation lime stamp; 
computer readable program code means for adjusting the 

oscillator clock time; and 
computer readable program code means for adding the 30 

adjusted theoretical presentation time stamp and the 

adjusted oscillator clock time. 

32. The computer program product according to claim 31, 
wherein said computer readable program code means for 
selecting a fine adjustment mode comprises: 35 

computer readable program code means for adjusting the 

theoretical presentation time stamp; 
computer readable program code means for adjusting the 

oscillator clock time; and 

' 40 

computer readable program code means for adding the 
adjusted theoretical presentation time stamp and the 
adjusted oscillator clock time. 



33. The computer program product according to claim 32, 
wherein said computer readable program code means for 
adjusting the theoretical presentation time stamp comprises 
the step of multiplying the theoretical presentation time 
stamp by a first compensation factor. 

34. The computer program product according to claim 33, 
wherein the first compensation factor is equal to about 0.9. 

35. The computer program product according to claim 32, 
wherein said computer readable program code means for 
adjusting the oscillator clock time comprises the step of 
multiplying the oscillator clock time by a second compen- 
sation factor. 

36. The computer program product according to claim 35, 
wherein the second compensation factor is equal to about 
0.1. 

37. A system for generating a real time multimedia data 
stream with accurate time stamps for decoding comprising: 

an first analog to digital converter that converts analog 
video frames to digitized video frames for encoding; 

a second analog to digital converter that converts analog 
audio data to digitized audio data for encoding; 

a video encoder for encoding the digitized video frames 
and providing an end of frame interrupt; 

an audio encoder for encoding digitized audio; 

a presentation time stamp counter for determining the 
theoretical present time stamp; 

an oscillator for clocking the encoded audio data, the 
audio analog to digital converter and providing a sys- 
tem clock; 

a presentation time stamp adjustor for providing an 
adjusted presentation time stamp which compensates 
for a difference between the theoretical frame rate 
corresponding to a video frame within the real time 
multimedia data stream and the oscillator clock; 

a presentation time stamp multiplexer for time stamping 
the encoded audio data with the oscillator and stamping 
the encoded video data with the adjusted presentation 
time stamp. 
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